Document both shapes for device resident outputs - #23102
Merged
shoumikhin merged 1 commit intoSep 24, 2026
Merged
Conversation
The CUDA page tells you to export device resident inputs and outputs with unplanned graph outputs. A Python caller cannot follow that. An unplanned output means the program reserves no buffer, so the caller has to supply one, and the Python bindings supply host memory. A CUDA delegated method then fails to run. The setting the page warns against is the one that works from Python. With non CPU memory planning on, a planned output buffer is allocated on the device, and skipping the device to host copy means nothing brings it back, so the result stays on the device and the caller receives a pointer to the runtime's own buffer. That is also what the exporters in this repository already do. The input half of the recipe was correct and is unchanged. The output half now says there are two shapes: planned when the caller cannot supply device memory, which is every Python caller today, and unplanned when it can, which today means a C++ caller handing the runtime a device tensor. Test plan: read against the example exporters in the repository, which use unplanned inputs and planned outputs with both copy skips.
🔗 Helpful Links🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/23102
Note: Links to docs will display an error until the docs builds have been completed. ❌ 1 New FailureAs of commit fc37994 with merge base b253af8 ( NEW FAILURE - The following job has failed:
This comment was automatically generated by Dr. CI and updates every 15 minutes. |
meta-cla
Bot
added
the
CLA Signed
Gasoonjia
approved these changes
Sep 24, 2026
nil-is-all
added
module: doc
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The CUDA page tells you to export device resident inputs and outputs with unplanned graph outputs.
A Python caller cannot follow that recipe.
An unplanned output means the program reserves no buffer, so the caller has to supply one before the
method runs. From C++ that is natural, since the caller makes a tensor on the device and hands over
its pointer. The Python bindings do it for the caller and they do it on the host, so a device
resident output is given host memory and the run fails.
The setting the page warns against is the one that works from Python. With
enable_non_cpu_memory_planningon, a planned output buffer is allocated on the device, andskip_d2h_for_method_outputsmeans nothing copies it back, so the result stays on the device andthe caller receives a pointer to the runtime's own buffer. That buffer belongs to the method and is
reused by the next execution, which the page now says. It is also what the exporters in this
repository already do, for example
examples/models/gemma4_31b/export.py.The input half of the recipe was correct and is unchanged. The output half now describes two shapes:
planned when the caller cannot supply device memory, which is every Python caller today, and
unplanned when it can, which today means a C++ caller handing the runtime a device tensor through
Module::set_output.Fixes #23101
Test plan: read against the example exporters in this repository, which use unplanned inputs and
planned outputs together with both copy skips. No code changed.
cc @mergennachin @nil-is-all @Gasoonjia @digantdesai